Testing and verification of modern software-based systems require automating both the test generation and the test execution. Nevertheless, present test development techniques and tests systems still rely on the expertise of a single engineer to efficiently and thoroughly prepare these tests.
Testing Applications-Under-Test ("AUTs") has been systematically and manually approached to counter three major challenges in validation and verification of AUTs for target systems.
First, high quality must be maintained even though software complexity increases. For example, telecommunication systems must have a downtime of less than minutes per year. This performance standard requires test coverage to be correspondingly high in order to ensure that latent software defects do not cause system failures after the product is deployed in the field.
Second, competitive pressures require manufacturers to have a quick turn-around time without compromising product reliability. Therefore, while testing must be expanded to improve coverage and to guarantee reliability, the present time requirements to conduct testing must be reduced.
Third, testing costs need to be minimized without sacrificing product quality. Thus, hiring additional personnel will not achieve this goal because of the additional costs.
Accordingly, software testing and verification for software-based systems has become a major hurdle to the delivery of new telecommunication products. Most of the verification time is spent in test generation, not testing.
Manual test generation, or script writing, is time consuming because it requires a laborious, brute-force approach. Working from the target system specifications, an engineer must try to conceive of every possible function and condition to which the AUT can be subjected, and then write a testcase to replicate the conditions. There is no way to ensure that all the functions and conditions are covered. Furthermore, if any changes are made later in the target system specifications, portions of the testcase may become obsolete due to its static nature. That is, testcases are unable to dynamically conform to changes in the system specifications, requiring an entire re-write of the testcase.
Clearly, the adequacy of the generated testcase relies on the expertise and experience of the engineer. Modeling tools that implement visual programming have become available to test engineers.
But such tools remain merely as aids to the test engineer. Consistent, thorough tests are not available because the engineer must still define the functions and conditions for the product model. The more experienced and weathered the engineer, the greater the depth of completeness and functionality of the resulting generated testcase. These qualities still remain with the engineer. If an experienced engineer leaves, then testcase generation again becomes a major hurdle to the delivery of new telecommunication products while a junior engineer becomes sufficiently trained to generate thorough and complete tests.
Also, such tools have embedded boundary conditions in the system model that are not readily redefined. The ability to selectively manipulate boundary conditions provides the engineer with insights to modify an AUT to overcome problematic software code portions.
Accordingly, a need exists for testcase generation of software-based computer systems that can provide thorough and consistent quality test scripts independent of the experience and knowledge of the test engineer. The testcase generation system is needed to allow a user to operate on a cross-section of an entire system model of a product to provide testcases. Also, a need exists for testcase script generation based on the accumulated knowledge of numerous test engineers such that the knowledge and experience remains, even after the departure of personnel.